Papers with recurrent neural networks
Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality (P18-2)
Copied to clipboard
| Challenge: | Recurrent neural tensor networks (RNNs) increase capacity by augmenting the size of the hidden layer, with significant increase in computational cost. |
| Approach: | They propose restricted recurrent neural tensor networks (r-RNTNs) which reserve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights . |
| Outcome: | The proposed model outperforms unrestricted RNTNs using only a small fraction of the parameters of unrestrained RNNNs. |
Transformer-Based Direct Hidden Markov Model for Machine Translation (2021.acl-srw)
Copied to clipboard
| Challenge: | Recent studies have found that word alignments produced by the multi-head cross-attention weights are poor. |
| Approach: | They propose to introduce the hidden Markov model to the transformer architecture and introduce alignment components while keeping the system monolithic. |
| Outcome: | The proposed model outperforms the baseline model but is slower in training and decoding. |
Computational Expressivity of Neural Language Models (2024.acl-tutorials)
Copied to clipboard
| Challenge: | Language models (LMs) are at the forefront of NLP research due to their versatility across diverse tasks. |
| Approach: | This tutorial will provide a framework for formal analysis of modern language models using tools from formal language theory. |
| Outcome: | This tutorial will provide a framework for formal analysis of modern language models using tools from formal language theory (FLT). |
An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networks (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing protection schemes for deep neural network models protect intellectual property rights from being abused, stolen and plagiarized. |
| Approach: | They propose a practical approach for the IPR protection on recurrent neural networks without all the bells and whistles of existing IPR solutions. |
| Outcome: | The proposed approach is robust and effective against ambiguity and removal attacks on different RNN variants. |
Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features (N19-2)
Copied to clipboard
Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei, Jimmy Lin
| Challenge: | e-commerce companies often have the option of escalating complaints by filing grievances with a government authority . this is detrimental to an ecommerce company, but this problem is challenging to solve by integrating recurrent neural networks with manually-engineered features. |
| Approach: | They propose a model that integrates recurrent neural networks with manually-engineered features to identify cases where the customer expresses such an intent. |
| Outcome: | The proposed model outperforms baseline models and provides better recall and triage for specialized agents. |
Natural Language Generation by Hierarchical Decoding with Linguistic Patterns (N18-2)
Copied to clipboard
| Challenge: | a common and mostly adopted method is the rule-based (or template-based) method for natural language generation. |
| Approach: | They propose a hierarchical decoding NLG model based on linguistic patterns in different levels. |
| Outcome: | The proposed method outperforms the traditional one with a smaller model size. |
Learning to Search in Long Documents Using Document Structure (C18-1)
Copied to clipboard
| Challenge: | Reading comprehension models are dominated by recurrent neural networks (RNNs) as documents become longer and questions become complex, sequential reading becomes a significant bottleneck. |
| Approach: | They propose a reading comprehension framework that uses document trees to model an agent that interleaves quick navigation with more expensive answer extraction. |
| Outcome: | The proposed model improves question answering performance compared to existing models and has a strong information-retrieval baseline. |
Multimodal Language Analysis with Recurrent Multistage Fusion (D18-1)
Copied to clipboard
| Challenge: | Comprehending multimodal language requires modeling interactions between modalities and between them. |
| Approach: | They propose a multistage fusion network which decomposes the fusion problem into multiple stages, each focused on a subset of multimodal signals for specialized, effective fusion. |
| Outcome: | The proposed model performs state-of-the-art across three datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition. |
How does Attention Affect the Model? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the effectiveness of attention in NLP do not consider changes in semantic capability of different components. |
| Approach: | They propose a framework that exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
| Outcome: | The proposed framework exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
Multimodal Modeling of Task-Mediated Confusion (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing models for confusion detection are under-explored, but they can be used to detect it computationally. |
| Approach: | They build upon prior work to develop models that detect confusion from three modalities: video, audio, and text. |
| Outcome: | The proposed models can detect confusion from facial expressions, prosody, and transcribed spoken language. |
Letting Emotions Flow: Success Prediction by Modeling the Flow of Emotions in Books (N18-2)
Copied to clipboard
| Challenge: | We obtained the best weighted F1-score of 69% for predicting books’ success in a multitask setting. |
| Approach: | They propose to model the flow of emotions over a book using recurrent neural networks and quantify its usefulness in predicting success in books. |
| Outcome: | The proposed model obtained the best weighted F1-score of 69% for predicting books’ success in a multitask setting. |
On Orthogonality Constraints for Transformers (2021.acl-short)
Copied to clipboard
Aston Zhang, Alvin Chan, Yi Tay, Jie Fu, Shuohang Wang, Shuai Zhang, Huajie Shao, Shuochao Yao, Roy Ka-Wei Lee
| Challenge: | a dedicated study on orthogonality constraints for transformers has been lacking . plug-and-play constraints increase the BLEU of transformers . |
| Approach: | They propose to use plug-and-play constraints to encourage matrices to be orthogonal for numerical stability. |
| Outcome: | The proposed constraint increases the BLEU on the large-scale WMT’16 EnDe benchmark by a factor of 28.4 to 29.6. |
Scheduled Sampling for Transformers (P19-2)
Copied to clipboard
| Challenge: | Existing studies show that scheduled sampling can be applied to recurrent neural networks to avoid exposure bias. |
| Approach: | They propose to use teacher forced embeddings and model predictions to avoid exposure bias in sequence-to-sequence generation. |
| Outcome: | The proposed technique achieves performance close to a teacher-forcing baseline on two language pairs and is promising for future research. |
Deep Neural Models for Medical Concept Normalization in User-Generated Texts (P19-2)
Copied to clipboard
| Challenge: | a medical concept normalization problem is a challenge since social media texts are ambiguous and noisy . a recent study shows that neural architectures leverage the semantic meaning of the entity mention . |
| Approach: | They propose to map a health-related entity mention to a controlled vocabulary . they use powerful neural networks and contextualized word representation models . |
| Outcome: | The proposed model outperforms existing state-of-the-art models in mapping medical concepts to medical terms . the proposed model is based on recurrent neural networks and contextualized word representation models . |
From Language to Language-ish: How Brain-Like is an LSTM’s Representation of Nonsensical Language Stimuli? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | LSTMs are often used to measure event related potentials, but are they able to generalize to new data in a human-like way? |
| Approach: | They asked whether an LSTM model represents a language sample with degraded semantic or syntactic information and whether it resembles the brain's reaction to the stimuli. |
| Outcome: | The results suggest that LSTMs and human brain handle nonsensical data similarly. |
Convolutional Neural Networks with Recurrent Neural Filters (D18-1)
Copied to clipboard
| Challenge: | Convolutional neural networks (CNNs) use recurrent neural networks as convolution filters to capture language compositionality and long-term dependencies. |
| Approach: | They propose to use recurrent neural networks (RNNs) as convolution filters to capture language compositionality and long-term dependencies. |
| Outcome: | The proposed convolutional neural networks achieve state-of-the-art on two sentences and the Stanford Sentiment Treebank. |
Linguistic Knowledge and Transferability of Contextual Representations (N19-1)
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
Self-Attentional Models for Lattice Inputs (P19-1)
Copied to clipboard
| Challenge: | Existing work has extended recurrent neural networks to model lattice inputs but these models suffer from slow computation speeds. |
| Approach: | They propose to extend the paradigm of self-attention to handle lattice inputs by adding probabilistic reachability masks that incorporate latticae structure into the model and support lattics if available. |
| Outcome: | The proposed model outperforms baseline models while being much faster to compute than previous models. |
Neural Sequence Segmentation as Determining the Leftmost Segments (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to segment sentences are mostly at token level, limiting their full potential to capture long-term dependencies. |
| Approach: | They propose a framework that incrementally segments natural language sentences at segment level. |
| Outcome: | The proposed framework outperforms baseline methods on syntactic chunking and Chinese part-of-speech tagging datasets. |
Implicit n-grams Induced by Recurrence (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that self-attention based models have limitations on modeling sequential transformations. |
| Approach: | They propose to extract some explainable features from trained RNNs that are reminiscent of classical n-grams features. |
| Outcome: | The proposed models can model interesting linguistic phenomena such as negation and intensification. |
Ab Initio: Automatic Latin Proto-word Reconstruction (C18-1)
Copied to clipboard
| Challenge: | Existing methods for proto-word reconstruction are time-consuming and manual, but few studies have done it . a recent study used cognates to reconstruct ancient languages from their modern counterparts . |
| Approach: | They propose to use Latin proto-words to automate the process of proto-language reconstruction . they leverage information from all modern languages and use conditional random fields for sequence labeling . |
| Outcome: | The proposed method improves on previous results and requires less data . it is based on word forms in multiple Romance languages and on recurrent neural networks . |
Contextual String Embeddings for Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | Recent advances in language modeling have made it viable to model language as distributions over characters. |
| Approach: | They propose to leverage internal states of a trained character language model to produce a new type of word embeddings. |
| Outcome: | The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks. |
On the Robustness of Self-Attentive Models (P19-1)
Copied to clipboard
| Challenge: | Experimental results show that self-attentive neural models are more robust against adversarial perturbations compared to recurrent neural networks. |
| Approach: | They propose an adversarial attack algorithm that generates more natural adversarials . they propose to use the attention mechanism to learn a context-dependent representation . |
| Outcome: | The proposed attack algorithm generates more natural adversarial examples that could mislead models but not humans. |
Rational Recurrences (D18-1)
Copied to clipboard
| Challenge: | Recent studies show that neural models lack strong intuitions . recent studies show connections between convolutional neural networks and weighted finite state automata (WFSAs) |
| Approach: | They show that some recurrent neural networks share a connection to weighted finite state automata (WFSAs) they define rational recurrences as recursive hidden state update functions . they propose to use these functions to write forward calculations of a finite set of WFSA's . |
| Outcome: | The proposed model outperforms two baselines on language modeling and text classification. |
Global-Local Modeling with Prompt-Based Knowledge Enhancement for Emotion Inference in Conversation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on emotion recognition focus on recognizing emotions through a speaker’s utterance, while research on emotion inference predicts emotions of addressees through previous utterations. |
| Approach: | They propose a global-local modeling method based on recurrent neural networks and pre-trained language models to do emotion inference in conversation. |
| Outcome: | The proposed method achieves state-of-the-art on three datasets. |
A Real-World Data Resource of Complex Sensitive Sentences Based on Documents from the Monsanto Trial (2020.lrec-1)
Copied to clipboard
| Challenge: | Sensitive information detection is of great importance in a number of applications where unintended leaks of sensitive information may incur severe negative consequences. |
| Approach: | They propose to use a corpus of sentences to evaluate sensitive information detection approaches . they employ human annotations and automatically infer labels from domain experts . |
| Outcome: | The proposed models are based on a monsanto trial and are evaluated on sentence level. |
Meaning to Form: Measuring Systematicity as Information (P19-1)
Copied to clipboard
| Challenge: | A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between word forms and their meaning, or does some systematic phenomenon pervade? |
| Approach: | They propose to quantify the systematicity of the sign using mutual information and recurrent neural networks to examine 106 languages. |
| Outcome: | The proposed model reduces entropy in a word form conditioned on its semantic representation and recovers English examples of systematic affixes. |
VICTOR: a Dataset for Brazilian Legal Documents Classification (2020.lrec-1)
Copied to clipboard
Pedro Henrique Luz de Araujo, Teófilo Emídio de Campos, Fabricio Ataides Braz, Nilton Correia da Silva
| Challenge: | Approximately 10% of these are unstructured and requiring a lot of time to sort through. |
| Approach: | They propose to use a dataset built from Brazil's Supreme Court digitalized legal documents to improve document type classification and theme assignment tasks. |
| Outcome: | The proposed dataset is based on 45 thousand appeals and contains roughly 692 thousand documents—about 4.6 million pages. |
Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog (N18-1)
Copied to clipboard
| Challenge: | Existing approaches to reading comprehension on multiparty dialogs have focused on children's stories or newswire. |
| Approach: | They propose a new corpus and a robust deep learning architecture for a task in reading comprehension on multiparty dialog. |
| Outcome: | The proposed model outperforms the state-of-the-art model on a different genre using bidirectional LSTM, showing a 13.0+% improvement for longer dialogs. |
Contextual Dependencies in Time-Continuous Multidimensional Affect Recognition (L18-1)
Copied to clipboard
| Challenge: | despite of the research done in this area there is still no agreement on this issue. |
| Approach: | a paper compares the amount of context used in a model and performance of a time-continuous labelled spontaneous interaction. |
| Outcome: | a new study shows that the amount of context used in a model and performance is similar across models . the results show that knowledge about an appropriate context can reduce complexity and flexibility . |
Recurrent One-Hop Predictions for Reasoning over Knowledge Graphs (C18-1)
Copied to clipboard
| Challenge: | Large scale knowledge graphs (KGs) such as Freebase are generally incomplete. |
| Approach: | They propose a model that predicts entities at each step of mh-KB paths . the model is based on recurrent neural networks and vector representations of entities and relations . |
| Outcome: | The proposed models show state-of-the-art for two important multi-hop KG reasoning tasks. |
Recurrent Neural Networks as Weighted Language Recognizers (N18-1)
Copied to clipboard
| Challenge: | Recent experiments show that RNNs outperform other methods in assigning high probability to held-out English text. |
| Approach: | They focus on the single-layer, ReLU-activation, rational-weight RNNs with softmax . they show that most problems for such RNN are undecidable . |
| Outcome: | The proposed model outperforms other methods in assigning high probability to held-out English text. |
Transformer-Exclusive Cross-Modal Representation for Vision and Language (2021.findings-acl)
Copied to clipboard
| Challenge: | a number of approaches to crossmodal representation have been used, but transformer architecture has taken over the recurrent neural networks in natural language processing tasks. |
| Approach: | They propose to use transformer architecture to handle cross-modal representations for vision and language with compatible performance to convolutional neural networks. |
| Outcome: | The proposed model outperforms recurrent neural networks in vision and language representations with transformer architecture. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings (P18-1)
Copied to clipboard
| Challenge: | recurrent neural networks have produced significant advances in part-of-speech tagging accuracy . a common feature of these models is the presence of rich initial word encodings . however, word or sub-word information interacts only through subsequent recursive layers . |
| Approach: | They propose to use recurrent neural networks with sentence-level context for initial character and word-based representations. |
| Outcome: | The proposed model has the highest accuracy of all participating systems in the CoNLL 2017 task. |
SiMFy: A Simple Yet Effective Approach for Temporal Knowledge Graph Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for temporal knowledge graph reasoning suffer from low training efficiency and insufficient generalization ability. |
| Approach: | They propose a temporal knowledge graph reasoning approach that uses multilayer perceptron to model the structural dependencies of events and adopts a fixed-frequency strategy to incorporate historical frequency during inference. |
| Outcome: | The proposed model achieves state-of-the-art performance with faster convergence speed and better generalization ability. |
A Survey of Retentive Network (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the effectiveness of the Retentive Networks have not yet been conducted. |
| Approach: | They propose a retention mechanism that integrates the inductive bias of recurrent neural networks with the parallelizable training advantages of attention-based models. |
| Outcome: | The proposed retention mechanism combines the inductive bias of recurrent neural networks with the parallelizable training advantages of attention-based models. |
Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks typically need large labeled data for training and are not easily interpretable. |
| Approach: | They propose a type of recurrent neural networks that combine neural networks and regular expression rules. |
| Outcome: | The proposed recurrent neural networks outperform previous neural approaches in low- and zero-shot scenarios and remain very competitive in rich-resource settings. |
An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for learning multi-word expressions have language sparsity and are not supervised. |
| Approach: | They propose an unsupervised approach to learning a compositional representation function for multi-word expressions . they use a Tratz dataset to train the composition function on the word-semantic relation . |
| Outcome: | The proposed method outperforms the previous state-of-the-art method on the Tratz dataset with an F1 score of 50.4%. |
An End-to-End Generative Architecture for Paraphrase Generation (D19-1)
Copied to clipboard
| Challenge: | Existing methods for generating paraphrases with linguistic knowledge are often domain specific and hard to scale, or yield inferior results. |
| Approach: | They propose an end-to-end conditional generative architecture for generating paraphrases via adversarial training which does not depend on extra linguistic information. |
| Outcome: | The proposed method outperforms existing models on automatic metrics and human evaluations on four public datasets. |
Bayesian Compression for Natural Language Processing (D18-1)
Copied to clipboard
| Challenge: | In natural language processing, recurrent neural networks have a huge number of parameters. |
| Approach: | They propose a Bayesian sparsification technique which allows compressing RNNs dozens or hundreds of times without time-consuming hyperparameters tuning. |
| Outcome: | The proposed technique compresses the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. |
On the Relationship Between RNN Hidden-State Vectors and Semantic Structures (2024.findings-acl)
Copied to clipboard
| Challenge: | Using hidden-state vectors of recurrent neural networks (RNNs) we examine the assumption that hidden- state vectors tend to form clusters of semantically similar vectors, which we dub the clustering hypothesis. |
| Approach: | They propose to use recurrent neural networks (RNNs) that model processes with internal states to test their hypothesis. |
| Outcome: | The proposed model is based on a set of RNNs that were trained to recognize regular languages and a context-free language. |
Assessing the Ability of Self-Attention Networks to Learn Word Order (P19-1)
Copied to clipboard
| Challenge: | Existing studies have attributed SAN to being weak at learning positional information for sequence modeling due to lack of recurrence structure. |
| Approach: | They propose a word reordering detection task to quantify how well word order information is learned by SAN and RNN. |
| Outcome: | The proposed task quantifies how well word order information learned by SAN and RNN is learned. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
On the Representational Capacity of Recurrent Neural Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on LMs as formal languages, but they do not consider language membership. |
| Approach: | They extend the Turing completeness result to the probabilistic case . they show that a rationally weighted RLM can simulate any deterministic Turing machine . |
| Outcome: | The proposed model can simulate any deterministic Turing machine with rationally weighted transitions . the proposed model is based on recurrent neural networks with a rational weighting over strings . |
Neural Disambiguation of Lemma and Part of Speech in Morphologically Rich Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | a method for disambiguating the lemma and part of speech of ambiguous words is proposed . a morphological analyser produces multiple analyses for ambiguously words . |
| Approach: | They propose a method for disambiguating the lemma and part of speech of ambiguous words in context . they use a large un-annotated corpus of text and a morphological analyser to train neural networks on the output of the analyser . |
| Outcome: | The proposed method outperforms the state-of-the-art on POS and lemma disambiguation in morphologically rich languages using no manual disambiguations or data annotations. |
Dilated Convolutional Neural Networks for Lightweight Diacritics Restoration (2022.lrec-1)
Copied to clipboard
| Challenge: | Diacritics restoration is a ubiquitous task in the Latin-alphabet-based English-dominated Internet language environment. |
| Approach: | They propose a 1D dilated convolution-based approach which operates on a character-level. |
| Outcome: | The proposed approach surpasses similar models and is competitive with larger models. |
Corpora for Document-Level Neural Machine Translation (2020.lrec-1)
Copied to clipboard
| Challenge: | Document-level machine translation models translate sentences in isolation, but there are three main problems for document-level models. |
| Approach: | They propose to use document-level machine translation to capture discourse dependencies across sentences by considering a document as a whole. |
| Outcome: | The proposed method captures discourse dependencies across sentences by considering a document as a whole. |
Learning Universal Sentence Representations with Mean-Max Attention Autoencoder (D18-1)
Copied to clipboard
| Challenge: | Existing methods to learn universal sentence representations focus on supervised learning. |
| Approach: | They propose a mean-max attention autoencoder that uses a multi-head mechanism to reconstruct the input sequence. |
| Outcome: | The proposed model outperforms state-of-the-art unsupervised single methods on a wide range of 10 transfer tasks. |
The Importance of Being Recurrent for Modeling Hierarchical Structure (D18-1)
Copied to clipboard
| Challenge: | Recent work shows that recurrent neural networks can implicitly capture hierarchical information when trained to solve common natural language processing tasks. |
| Approach: | They propose a convolutional sequence-to-sequence model that exploits hierarchical information implicitly. |
| Outcome: | The proposed model is recurrent and non-recurrent, and it can model hierarchical structure implicitly. |
Learning Sequence Encoders for Temporal Knowledge Graph Completion (D18-1)
Copied to clipboard
| Challenge: | Existing work on link prediction in knowledge graphs has focused on static multi-relational data. |
| Approach: | They propose to learn latent entity and relation type representations to incorporate temporal information into knowledge graphs. |
| Outcome: | The proposed approach is robust to common challenges in real-world KGs. |
Learning Word Representations with Cross-Sentence Dependency for End-to-End Co-reference Resolution (D18-1)
Copied to clipboard
| Challenge: | Existing word embedding models generate word representations by running long short-term memory recurrent neural networks on each sentence of an input article or conversation separately. |
| Approach: | They propose a word embedding model that learns cross-sentence dependency . they use linear sentence linking and attentional sentence linking to learn cross-entry dependency based on context sentences . |
| Outcome: | The proposed model improves end-to-end co-reference resolution by taking knowledge from context sentences and the entire document. |
Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | Currently, image retrieval systems can retrieve relevant results for diverse inputs, but they do not provide a way to intentionally inject variety into the search results. |
| Approach: | They propose a multimodal dataset that combines semantic annotations with image bounding boxes. |
| Outcome: | The proposed system improves image retrieval performance and flexibility. |
Czech Historical Named Entity Corpus v 1.0 (2020.lrec-1)
Copied to clipboard
| Challenge: | a lack of annotated historical data for named entity recognition is an obstacle to research in this area. |
| Approach: | They propose to create an annotated corpus for named entity recognition in historical documents . they define domain-specific named entity types and create an annotation manual . |
| Outcome: | The proposed corpus is available for research and is available to download . it is the first annotated historical corpus for named entity recognition (NER) |
Using Adversarial Examples in Natural Language Processing (L18-1)
Copied to clipboard
| Challenge: | Recent advances in machine learning have led to the use of adversarial examples in training of neural networks. |
| Approach: | They investigate the effect of using adversarial examples during training of recurrent neural networks whose text input is in the form of a sequence of word/character embeddings. |
| Outcome: | The proposed method provides regularization effect and enables training of models with greater number of parameters without overfitting. |
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions. |
| Approach: | They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed . |
| Outcome: | The proposed extensions overcome limitations of the LT and self-referential weight matrices. |
Embedding Dynamic Attributed Networks by Modeling the Evolution Processes (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to embed nodes into low-dimensional vectors focus on static networks, but in practice, many networks are evolving over time and hence are dynamic, e.g., social networks. |
| Approach: | They propose to extract high-order neighborhood information at each given timestamp and then use an embedding prediction framework to capture the temporal correlations. |
| Outcome: | Extensive experiments on four real-world datasets show that the proposed method outperforms baseline methods for dynamic link prediction and node classification tasks. |
MCˆ2: Multi-perspective Convolutional Cube for Conversational Machine Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Existing models combine previous questions for conversation understanding and only employ recurrent neural networks (RNN) for reasoning. |
| Approach: | They propose a multi-perspective convolutional cube model that integrates 1D and 2D convolutions with recurrent neural networks (RNN) to understand context from different perspectives. |
| Outcome: | The proposed model is based on the Conversational Question Answering (CoQA) dataset and achieves state-of-the-art results. |
Finetuning Pretrained Transformers into RNNs (2021.emnlp-main)
Copied to clipboard
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith
| Challenge: | Efficient transformers outperform recurrent neural networks in natural language generation, but this comes with significant computational cost and memory footprint during generation. |
| Approach: | They propose to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy. |
| Outcome: | The proposed transformers outperform recurrent neural networks in natural language generation but come with significant computational and memory footprint during generation. |
RWKV: Reinventing RNNs for the Transformer Era (2023.findings-emnlp)
Copied to clipboard
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu
| Challenge: | recurrent neural networks struggle to match the performance of Transformers due to limitations in parallelization and scalability. |
| Approach: | They propose a model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. |
| Outcome: | The proposed model performs on par with similarly sized RNNs, suggesting future work can leverage this architecture to create more efficient models. |